{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LeNet Lab Solution\n",
    "![LeNet Architecture](lenet.png)\n",
    "Source: Yan LeCun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "Load the MNIST data, which comes pre-loaded with TensorFlow.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-1-743343216c71>:3: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From C:\\Program Files\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From C:\\Program Files\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From C:\\Program Files\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From C:\\Program Files\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "\n",
      "Image Shape: (28, 28, 1)\n",
      "\n",
      "Training Set:   55000 samples\n",
      "Validation Set: 5000 samples\n",
      "Test Set:       10000 samples\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", reshape=False)\n",
    "X_train, y_train           = mnist.train.images, mnist.train.labels\n",
    "X_validation, y_validation = mnist.validation.images, mnist.validation.labels\n",
    "X_test, y_test             = mnist.test.images, mnist.test.labels\n",
    "\n",
    "assert(len(X_train) == len(y_train))\n",
    "assert(len(X_validation) == len(y_validation))\n",
    "assert(len(X_test) == len(y_test))\n",
    "\n",
    "print()\n",
    "print(\"Image Shape: {}\".format(X_train[0].shape))\n",
    "print()\n",
    "print(\"Training Set:   {} samples\".format(len(X_train)))\n",
    "print(\"Validation Set: {} samples\".format(len(X_validation)))\n",
    "print(\"Test Set:       {} samples\".format(len(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated Image Shape: (32, 32, 1)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Pad images with 0s\n",
    "X_train      = np.pad(X_train, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
    "X_validation = np.pad(X_validation, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
    "X_test       = np.pad(X_test, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
    "    \n",
    "print(\"Updated Image Shape: {}\".format(X_train[0].shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "X_train, y_train = shuffle(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "EPOCHS = 10\n",
    "BATCH_SIZE = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SOLUTION: Implement LeNet-5\n",
    "Implement the [LeNet-5](http://yann.lecun.com/exdb/lenet/) neural network architecture.\n",
    "\n",
    "This is the only cell you need to edit.\n",
    "### Input\n",
    "The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.\n",
    "\n",
    "### Architecture\n",
    "**Layer 1: Convolutional.** The output shape should be 28x28x6.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 14x14x6.\n",
    "\n",
    "**Layer 2: Convolutional.** The output shape should be 10x10x16.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 5x5x16.\n",
    "\n",
    "**Flatten.** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using `tf.contrib.layers.flatten`, which is already imported for you.\n",
    "\n",
    "**Layer 3: Fully Connected.** This should have 120 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 4: Fully Connected.** This should have 84 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 5: Fully Connected (Logits).** This should have 10 outputs.\n",
    "\n",
    "### Output\n",
    "Return the result of the 2nd fully connected layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.contrib.layers import flatten\n",
    "\n",
    "def LeNet(x):    \n",
    "    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer\n",
    "    mu = 0\n",
    "    sigma = 0.1\n",
    "    \n",
    "    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
    "    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma))\n",
    "    conv1_b = tf.Variable(tf.zeros(6))\n",
    "    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "\n",
    "    # SOLUTION: Activation.\n",
    "    conv1 = tf.nn.relu(conv1)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
    "    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.\n",
    "    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))\n",
    "    conv2_b = tf.Variable(tf.zeros(16))\n",
    "    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    conv2 = tf.nn.relu(conv2)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
    "    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.\n",
    "    fc0   = flatten(conv2)\n",
    "    \n",
    "    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
    "    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))\n",
    "    fc1_b = tf.Variable(tf.zeros(120))\n",
    "    fc1   = tf.matmul(fc0, fc1_W) + fc1_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc1    = tf.nn.relu(fc1)\n",
    "\n",
    "    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
    "    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))\n",
    "    fc2_b  = tf.Variable(tf.zeros(84))\n",
    "    fc2    = tf.matmul(fc1, fc2_W) + fc2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc2    = tf.nn.relu(fc2)\n",
    "\n",
    "    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 10.\n",
    "    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 10), mean = mu, stddev = sigma))\n",
    "    fc3_b  = tf.Variable(tf.zeros(10))\n",
    "    logits = tf.matmul(fc2, fc3_W) + fc3_b\n",
    "    \n",
    "    return logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (None, 32, 32, 1))\n",
    "y = tf.placeholder(tf.int32, (None))\n",
    "one_hot_y = tf.one_hot(y, 10)\n",
    "rate = 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-6-c6d1a678c49a>:59: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See `tf.nn.softmax_cross_entropy_with_logits_v2`.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "#logits = LeNet(x)\n",
    "\n",
    "\n",
    "# Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer\n",
    "mu = 0\n",
    "sigma = 0.1\n",
    "\n",
    "# SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
    "conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma))\n",
    "conv1_b = tf.Variable(tf.zeros(6))\n",
    "conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "\n",
    "# SOLUTION: Activation.\n",
    "conv1 = tf.nn.relu(conv1)\n",
    "\n",
    "# SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
    "conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "# SOLUTION: Layer 2: Convolutional. Output = 10x10x16.\n",
    "conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))\n",
    "conv2_b = tf.Variable(tf.zeros(16))\n",
    "conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "\n",
    "# SOLUTION: Activation.\n",
    "conv2 = tf.nn.relu(conv2)\n",
    "\n",
    "# SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
    "conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "# SOLUTION: Flatten. Input = 5x5x16. Output = 400.\n",
    "fc0   = flatten(conv2)\n",
    "\n",
    "# SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
    "fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))\n",
    "fc1_b = tf.Variable(tf.zeros(120))\n",
    "fc1   = tf.matmul(fc0, fc1_W) + fc1_b\n",
    "\n",
    "# SOLUTION: Activation.\n",
    "fc1    = tf.nn.relu(fc1)\n",
    "\n",
    "# SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
    "fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))\n",
    "fc2_b  = tf.Variable(tf.zeros(84))\n",
    "fc2    = tf.matmul(fc1, fc2_W) + fc2_b\n",
    "\n",
    "# SOLUTION: Activation.\n",
    "fc2    = tf.nn.relu(fc2)\n",
    "\n",
    "# SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 10.\n",
    "fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 10), mean = mu, stddev = sigma))\n",
    "fc3_b  = tf.Variable(tf.zeros(10))\n",
    "logits = tf.matmul(fc2, fc3_W) + fc3_b\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)\n",
    "loss_operation = tf.reduce_mean(cross_entropy)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = rate)\n",
    "training_operation = optimizer.minimize(loss_operation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))\n",
    "accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "def evaluate(X_data, y_data):\n",
    "    num_examples = len(X_data)\n",
    "    total_accuracy = 0\n",
    "    sess = tf.get_default_session()\n",
    "    for offset in range(0, num_examples, BATCH_SIZE):\n",
    "        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]\n",
    "        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "        total_accuracy += (accuracy * len(batch_x))\n",
    "    return total_accuracy / num_examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "\n",
      "EPOCH 1 ...\n",
      "Validation Accuracy = 0.969\n",
      "\n",
      "EPOCH 2 ...\n",
      "Validation Accuracy = 0.979\n",
      "\n",
      "EPOCH 3 ...\n",
      "Validation Accuracy = 0.982\n",
      "\n",
      "EPOCH 4 ...\n",
      "Validation Accuracy = 0.988\n",
      "\n",
      "EPOCH 5 ...\n",
      "Validation Accuracy = 0.985\n",
      "\n",
      "EPOCH 6 ...\n",
      "Validation Accuracy = 0.988\n",
      "\n",
      "EPOCH 7 ...\n",
      "Validation Accuracy = 0.986\n",
      "\n",
      "EPOCH 8 ...\n",
      "Validation Accuracy = 0.990\n",
      "\n",
      "EPOCH 9 ...\n",
      "Validation Accuracy = 0.988\n",
      "\n",
      "EPOCH 10 ...\n",
      "Validation Accuracy = 0.987\n",
      "\n",
      "Model saved\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    num_examples = len(X_train)\n",
    "    \n",
    "    print(\"Training...\")\n",
    "    print()\n",
    "    for i in range(EPOCHS):\n",
    "        X_train, y_train = shuffle(X_train, y_train)\n",
    "        for offset in range(0, num_examples, BATCH_SIZE):\n",
    "            end = offset + BATCH_SIZE\n",
    "            batch_x, batch_y = X_train[offset:end], y_train[offset:end]\n",
    "            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "            \n",
    "        validation_accuracy = evaluate(X_validation, y_validation)\n",
    "        print(\"EPOCH {} ...\".format(i+1))\n",
    "        print(\"Validation Accuracy = {:.3f}\".format(validation_accuracy))\n",
    "        print()\n",
    "        \n",
    "    saver.save(sess, './lenet')\n",
    "    print(\"Model saved\")\n",
    "    \n",
    "    # print weight and  bias\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from .\\lenet\n",
      "Test Accuracy = 0.989\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "\n",
    "    test_accuracy = evaluate(X_test, y_test)\n",
    "    print(\"Test Accuracy = {:.3f}\".format(test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from .\\lenet\n",
      "weight1:  (5, 5, 1, 6)\n",
      "weight2:  (5, 5, 6, 16)\n",
      "bias2:  (16,)\n",
      "weight_fc1:  (400, 120)\n",
      "bias_fc1:  (120,)\n",
      "fc1_re:  (1, 120)\n",
      "weight_fc2:  (120, 84)\n",
      "bias_fc2:  (84,)\n",
      "fc2_re:  (1, 84)\n",
      "weight_fc3:  (84, 10)\n",
      "bias_fc3:  (10,)\n",
      "fc3_re:  (1, 10)\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(threshold = 1e6)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "    #print(\"logits\", sess.run(logits, feed_dict={x: X_test[0:1]}))\n",
    "   \n",
    "\n",
    "    flog = open(\"weight1.log\", 'w')\n",
    "    \n",
    "    weight1 = sess.run(conv1_W)\n",
    "    print(\"weight1: \", weight1.shape) \n",
    "    #print(\"weight1: \", weight1) \n",
    "    \n",
    "    for i in range(5):\n",
    "        for j in range(5):\n",
    "            for o in range(6):\n",
    "                if (weight1[i][j][0][o]*32768).astype(int) < 0 :\n",
    "                    print(\"-\", file = flog, end='')\n",
    "                #print(\"16'd%d, \"%(weight1[i][j][0][o]*256).astype(int) , file = flog, end=' ')\n",
    "                print(\"16'd%d, \"%abs((weight1[i][j][0][o]*32768).astype(int)) , file = flog, end=' ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)\n",
    "        \n",
    "    print(\"==========================conv1_bias==========================\", file = flog)        \n",
    "    bias1 = sess.run(conv1_b)\n",
    "    print(\"\", file = flog)   \n",
    "    for o in range(6):\n",
    "        if (bias1[o]*32768*256).astype(int) < 0 :\n",
    "            print(\"-\", file = flog, end='')\n",
    "        print(\"24'd%d, \"%abs((bias1[o]*32768*256).astype(int)), file = flog, end=' ')\n",
    "        \n",
    "    flog.close()\n",
    "    \n",
    "\n",
    "####################################################    \n",
    "\n",
    "    flog = open(\"weight2.log\", 'w')\n",
    "    \n",
    "    weight2 = sess.run(conv2_W)\n",
    "    print(\"weight2: \", weight2.shape)   \n",
    "    \n",
    "    for row in range(5):\n",
    "        for col in range(5):\n",
    "            for o in range(16):\n",
    "                for i in range(6):\n",
    "                    if (weight2[row][col][i][o]*32768).astype(int) < 0 :\n",
    "                        print(\"-\", file = flog, end='')\n",
    "                    print(\"16'd%d, \"%abs((weight2[row][col][i][o]*32768).astype(int)) , file = flog, end=' ')\n",
    "                print(\"\", file = flog)\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)\n",
    "    \n",
    "    print(\"\", file = flog)  \n",
    "    \n",
    "    bias2 = sess.run(conv2_b)\n",
    "    print(\"bias2: \", bias2.shape)\n",
    "    print(\"\", file = flog)\n",
    "    print(\"\", file = flog)   \n",
    "    print(\"==========================conv2_bias==========================\", file = flog) \n",
    "    for o in range(16):\n",
    "        if (bias2[o]*32768*256).astype(int) < 0 :\n",
    "            print(\"-\", file = flog, end='')\n",
    "        print(\"24'd%d, \"%abs((bias2[o]*32768*256).astype(int)), file = flog, end=' ')\n",
    "        \n",
    "        \n",
    "    flog.close()\n",
    "\n",
    "####################################################   \n",
    "    flog = open(\"fc1.log\", 'w')\n",
    "    weight_fc1  = sess.run(fc1_W)\n",
    "    \n",
    "    print(\"weight_fc1: \", weight_fc1.shape)\n",
    "    \n",
    "    for o in range(120):\n",
    "        for i in range(400):\n",
    "            #print( \"%2f, \"%weight_fc1[i][o], file = flog, end=' ')\n",
    "            #print(\"%6d, \"%(weight_fc1[i][o]).astype(int) , file = flog, end=' ')\n",
    "            if (weight_fc1[i][o]*32768).astype(int) < 0 :\n",
    "                print(\"-\", file = flog, end='')\n",
    "            print(\"16'd%d, \"%abs((weight_fc1[i][o]*32768).astype(int)) , file = flog, end=' ')\n",
    "            if i%16==15 :\n",
    "                print(\"\", file = flog) \n",
    "        print(\"\", file = flog)  \n",
    "    bias_fc1  = sess.run(fc1_b)\n",
    "    print(\"bias_fc1: \", bias_fc1.shape)\n",
    "    print(\"==========================bias_fc1==========================\", file = flog)   \n",
    "    for o in range(120):\n",
    "        if (bias_fc1[o]*32768*256).astype(int) < 0 :\n",
    "            print(\"-\", file = flog, end='')\n",
    "        print(\"24'd%d, \"%abs((bias_fc1[o]*32768*256).astype(int)) , file = flog, end=' ')\n",
    "        if o%16==15 :\n",
    "            print(\"\", file = flog)  \n",
    "    print(\"\", file = flog)   \n",
    "    \n",
    "    fc1_re = sess.run(fc1, feed_dict={x: X_test[0:1]})\n",
    "    print(\"fc1_re: \", fc1_re.shape)\n",
    "    print(\"==========================fc1_re==========================\", file = flog)   \n",
    "    \n",
    "    for i in range(120):\n",
    "        print(\"%6d, \"%(fc1_re[0][i]*256).astype(int), file = flog, end=' ')\n",
    "        if i%16==15 :\n",
    "            print(\"\", file = flog) \n",
    "    print(\"\", file = flog)  \n",
    "    \n",
    "    flog.close()\n",
    "#################################################### \n",
    "    flog = open(\"fc2.log\", 'w')\n",
    "    weight_fc2  = sess.run(fc2_W)\n",
    "    \n",
    "    print(\"weight_fc2: \", weight_fc2.shape)\n",
    "    \n",
    "    for o in range(84):\n",
    "        for i in range(120):\n",
    "            if (weight_fc2[i][o]*32768).astype(int) < 0 :\n",
    "                print(\"-\", file = flog, end='')\n",
    "            #print( \"%2f, \"%weight_fc1[i][o], file = flog, end=' ')\n",
    "            #print(\"%6d, \"%(weight_fc1[i][o]).astype(int) , file = flog, end=' ')\n",
    "            print(\"16'd%d, \"%abs((weight_fc2[i][o]*32768).astype(int)) , file = flog, end=' ')\n",
    "            if i%16==15 :\n",
    "                print(\"\", file = flog) \n",
    "        print(\"\", file = flog)  \n",
    "    bias_fc2  = sess.run(fc2_b)\n",
    "    print(\"bias_fc2: \", bias_fc2.shape)\n",
    "    print(\"==========================bias_fc2==========================\", file = flog)  \n",
    "    for o in range(84):\n",
    "        if (bias_fc2[o]*32768*256).astype(int) < 0 :\n",
    "            print(\"-\", file = flog, end='')\n",
    "        print(\"24'd%d, \"%abs((bias_fc2[o]*32768*256).astype(int)) , file = flog, end=' ')\n",
    "        if o%16==15 :\n",
    "            print(\"\", file = flog)  \n",
    "    print(\"\", file = flog)   \n",
    "    \n",
    "    fc2_re = sess.run(fc2, feed_dict={x: X_test[0:1]})\n",
    "    print(\"fc2_re: \", fc2_re.shape)\n",
    "    print(\"==========================fc2_re==========================\", file = flog)   \n",
    "    \n",
    "    for i in range(84):\n",
    "        print(\"%6d, \"%(fc2_re[0][i]*256).astype(int), file = flog, end=' ')\n",
    "        if i%16==15 :\n",
    "            print(\"\", file = flog) \n",
    "    print(\"\", file = flog)  \n",
    "    \n",
    "    flog.close()\n",
    "#################################################### \n",
    "    flog = open(\"fc3.log\", 'w')\n",
    "    weight_fc3  = sess.run(fc3_W)\n",
    "    \n",
    "    print(\"weight_fc3: \", weight_fc3.shape)\n",
    "    \n",
    "    for o in range(10):\n",
    "        for i in range(84):\n",
    "            if (weight_fc3[i][o]*32768).astype(int) < 0 :\n",
    "                print(\"-\", file = flog, end='')\n",
    "            #print( \"%2f, \"%weight_fc1[i][o], file = flog, end=' ')\n",
    "            #print(\"%6d, \"%(weight_fc1[i][o]).astype(int) , file = flog, end=' ')\n",
    "            print(\"16'd%d, \"%abs((weight_fc3[i][o]*32768).astype(int)) , file = flog, end=' ')\n",
    "            if i%16==15 :\n",
    "                print(\"\", file = flog) \n",
    "        print(\"\", file = flog)  \n",
    "    bias_fc3  = sess.run(fc3_b)\n",
    "    print(\"bias_fc3: \", bias_fc3.shape)\n",
    "    print(\"==========================bias_fc3==========================\", file = flog) \n",
    "    for o in range(10):\n",
    "        if (bias_fc3[o]*32768*256).astype(int) < 0 :\n",
    "            print(\"-\", file = flog, end='')\n",
    "        print(\"24'd%d, \"%abs((bias_fc3[o]*32768*256).astype(int)) , file = flog, end=' ')\n",
    "        if o%16==15 :\n",
    "            print(\"\", file = flog)  \n",
    "    print(\"\", file = flog)   \n",
    "    \n",
    "    fc3_re = sess.run(logits, feed_dict={x: X_test[0:1]})\n",
    "    print(\"fc3_re: \", fc3_re.shape)\n",
    "    print(\"==========================fc3_re==========================\", file = flog)   \n",
    "    \n",
    "    for i in range(10):\n",
    "        print(\"%6d, \"%(fc3_re[0][i]*256).astype(int), file = flog, end=' ')\n",
    "        if i%16==15 :\n",
    "            print(\"\", file = flog) \n",
    "    print(\"\", file = flog)  \n",
    "    \n",
    "    flog.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from .\\lenet\n",
      "conv1:  (1, 28, 28, 6)\n",
      "relu1:  (?, 28, 28, 6)\n",
      "pool1_re:  (1, 14, 14, 6)\n",
      "conv2:  (1, 10, 10, 16)\n",
      "pool2_re:  (1, 5, 5, 16)\n",
      "fc0_re:  (1, 400)\n",
      "X_test:  (10000, 32, 32, 1)\n",
      "fc3_re:  (100, 10)\n",
      "y_test:  (10000,)\n"
     ]
    }
   ],
   "source": [
    "# test \n",
    "np.set_printoptions(threshold = 1e6)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "    \n",
    "        \n",
    "####################################################     \n",
    "    flog = open(\"conv1.log\", 'w')\n",
    "    \n",
    "    conv1_test   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "    relu1 = tf.nn.relu(conv1_test)\n",
    "    \n",
    "    conv_re = sess.run(conv1_test, feed_dict={x: X_test[0:1]})\n",
    "    relu1_re = sess.run(relu1, feed_dict={x: X_test[0:1]})\n",
    "    conv1\n",
    "    pool1_re = sess.run(conv1, feed_dict={x: X_test[0:1]})\n",
    "    print(\"conv1: \", conv_re.shape) \n",
    "    #print(\"conv1: \", conv_re) \n",
    "    print(\"relu1: \", relu1.shape) \n",
    "    print(\"pool1_re: \", pool1_re.shape) \n",
    "    \n",
    "    \n",
    "    print(\"conv1: \", conv_re.shape, file = flog) \n",
    "    #print(\"conv1: \", conv_re, file = flog) \n",
    "    for o in range(6):\n",
    "        for i in range(28):\n",
    "            for j in range(28):\n",
    "                print(conv_re[0][i][j][o], file = flog, end=' ')\n",
    "                #print(relu1_re[0][i][j][o], file = flog, end='     ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)\n",
    "    \n",
    "\n",
    "    print(\"\", file = flog)      \n",
    "    for i in range(28):\n",
    "        for j in range(28):\n",
    "            for o in range(6):\n",
    "                print(\"%6d, \"%(conv_re[0][i][j][o]*256).astype(int), file = flog, end=' ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)    \n",
    "        \n",
    "    print(\"==========================pool1_re==========================\", file = flog)     \n",
    "    \n",
    "    print(\"\", file = flog)      \n",
    "    for i in range(14):\n",
    "        for j in range(14):\n",
    "            for o in range(6):\n",
    "                print(\"%6d, \"%(pool1_re[0][i][j][o]*256).astype(int), file = flog, end=' ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)    \n",
    "    flog.close()\n",
    "    \n",
    "####################################################        \n",
    "    \n",
    "    conv2_in = tf.placeholder(tf.float32, (None, 14, 14, 6))\n",
    "    \n",
    "    ones32x32x6 = np.ones(14*14*6)\n",
    "    ones32x32x6 = ones32x32x6.reshape(1, 14,14,6)\n",
    "    #tensor32x32x6\n",
    "    \n",
    "    flog = open(\"conv2.log\", 'w')\n",
    "    \n",
    "    conv2_test   = tf.nn.conv2d(conv2_in, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    #conv2_test   = tf.nn.conv2d(conv2_in, conv2_W, strides=[1, 1, 1, 1], padding='VALID')\n",
    "    conv2_2d   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    \n",
    "    \n",
    "    #conv2_test   = tf.nn.conv2d(conv2_in, conv2_W, strides=[1, 1, 1, 1], padding='VALID')\n",
    "    conv2_re = sess.run(conv2_2d, feed_dict={x: X_test[0:1]})\n",
    "    #conv2_re = sess.run(conv2_test, feed_dict={conv2_in: ones32x32x6})\n",
    "    print(\"conv2: \", conv2_re.shape)    \n",
    "    \n",
    "    print(\"\", file = flog)      \n",
    "    for i in range(10):\n",
    "        for j in range(10):\n",
    "            for o in range(16):\n",
    "                print(\"%d, \"%(conv2_re[0][i][j][o]*256).astype(int), file = flog, end=' ')\n",
    "                #print(\"%6d, \"%(conv2_re[0][i][j][o]).astype(int), file = flog, end=' ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)    \n",
    "        \n",
    "    pool2_re = sess.run(conv2, feed_dict={x: X_test[0:1]})\n",
    "    print(\"==========================pool2_re==========================\", file = flog)     \n",
    "    print(\"pool2_re: \", pool2_re.shape)  \n",
    "    print(\"\", file = flog)      \n",
    "    for i in range(5):\n",
    "        for j in range(5):\n",
    "            for o in range(16):\n",
    "                print(\"%6d, \"%(pool2_re[0][i][j][o]*256).astype(int), file = flog, end=' ')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)   \n",
    "        \n",
    "        \n",
    "    fc0_re = sess.run(fc0, feed_dict={x: X_test[0:1]})\n",
    "    print(\"fc0_re: \", fc0_re.shape)  \n",
    "    print(\"==========================fc0_re==========================\", file = flog)     \n",
    "    \n",
    "    for i in range(400):\n",
    "        print(\"%6d, \"%(fc0_re[0][i]*256).astype(int), file = flog, end=' ')\n",
    "        if i%16==15 :\n",
    "            print(\"\", file = flog) \n",
    "    flog.close()      \n",
    "        \n",
    "        \n",
    "        \n",
    "####################################################        \n",
    "    \n",
    "    flog = open(\"test_digits.log\", 'w')\n",
    "    print(\"X_test: \", X_test.shape)\n",
    "    for n in range(100):\n",
    "        for i in range(32):\n",
    "            for j in range(32):\n",
    "                print(\"%4d, \"%(X_test[n][i][j][0]*255).astype(int), file = flog, end='')\n",
    "            print(\"\", file = flog)\n",
    "        print(\"\", file = flog)\n",
    "        print(\"\", file = flog)\n",
    "        \n",
    "    fc3_re = sess.run(logits, feed_dict={x: X_test[0:100]})\n",
    "    print(\"fc3_re: \", fc3_re.shape)\n",
    "    print(\"==========================fc3_re==========================\", file = flog)   \n",
    "    \n",
    "    for n in range(100):\n",
    "        for i in range(10):\n",
    "            print(\"%6d, \"%(fc3_re[n][i]*256).astype(int), file = flog, end=' ')\n",
    "        print(\"\", file = flog)\n",
    "        \n",
    "        \n",
    "    print(\"==========================digits==========================\", file = flog)   \n",
    "    y_test\n",
    "    print(\"y_test: \", y_test.shape)\n",
    "    for n in range(100):\n",
    "        print(\"%d, %d\"% (n, y_test[n]), file = flog, end=' ')\n",
    "        print(\"\", file = flog)  \n",
    "        \n",
    "    print(\"\", file = flog)  \n",
    "    \n",
    "    flog.close()\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
